{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "1. spaCy.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Afanasyy/colab/blob/main/1_spaCy.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 0. Библиотеки для NLP\n",
        "\n",
        "NLP - natural language processing (обработка естественного языка)\n",
        "\n",
        "## Зачем нужны библиотеки?\n",
        "\n",
        "\n",
        "*   Tokenization\n",
        "*   Part-of-speech (POS) Tagging\n",
        "*   Lemmatization\n",
        "*   Named Entity Recognition (NER)\n",
        "*   Similarity\n",
        "*   Text Classification\n",
        "*   etc.\n",
        "\n",
        "## Библиотеки для NLP\n",
        "\n",
        "\n",
        "*   NLTK: English\n",
        "*   **spaCy**: English, Spanish, Russian, etc.\n",
        "*   natasha: Russian\n",
        "*   etc.\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "so3CpK1ejqlN"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 1. spaCy: Введение\n",
        "\n",
        "https://spacy.io/"
      ],
      "metadata": {
        "id": "FfwSH_5HmUYq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Загрузка модели и установка модели"
      ],
      "metadata": {
        "id": "va0I5GM4zV_4"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Перед началом работы необходимо загрузить библиотеку на компьютер с помощью pip install (через cmd, jupyter notebook или гугл колаб). Также необходимо загрузить русскую языковую модель.\n",
        "\n",
        "Существует 2 русские модели:\n",
        "\n",
        "\n",
        "*   ru_core_news_sm - быстрая и небольшая, но менее точная.\n",
        "*   ru_core_news_lg - тяжелая и медленная, но точная.\n",
        "\n"
      ],
      "metadata": {
        "id": "6V9c5BbQqBiD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install spacy==3.2.4\n",
        "!python -m spacy download ru_core_news_sm"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zc9CdI-tj9bq",
        "outputId": "a54c1999-56e6-4f0d-b0cb-849c83eb7993"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: spacy==3.2.4 in /usr/local/lib/python3.7/dist-packages (3.2.4)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (3.0.6)\n",
            "Requirement already satisfied: wasabi<1.1.0,>=0.8.1 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (0.9.1)\n",
            "Requirement already satisfied: typing-extensions<4.0.0.0,>=3.7.4 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (3.10.0.2)\n",
            "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (3.3.0)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (21.3)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (2.23.0)\n",
            "Requirement already satisfied: typer<0.5.0,>=0.3.0 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (0.4.1)\n",
            "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (2.0.7)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (2.0.6)\n",
            "Requirement already satisfied: click<8.1.0 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (7.1.2)\n",
            "Requirement already satisfied: pathy>=0.3.5 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (0.6.1)\n",
            "Requirement already satisfied: srsly<3.0.0,>=2.4.1 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (2.4.3)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (1.8.2)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (1.0.6)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (4.64.0)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (57.4.0)\n",
            "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (1.21.5)\n",
            "Requirement already satisfied: thinc<8.1.0,>=8.0.12 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (8.0.15)\n",
            "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (1.0.2)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (0.4.1)\n",
            "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.8 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (3.0.9)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.7/dist-packages (from spacy==3.2.4) (2.11.3)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from catalogue<2.1.0,>=2.0.6->spacy==3.2.4) (3.8.0)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=20.0->spacy==3.2.4) (3.0.8)\n",
            "Requirement already satisfied: smart-open<6.0.0,>=5.0.0 in /usr/local/lib/python3.7/dist-packages (from pathy>=0.3.5->spacy==3.2.4) (5.2.1)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy==3.2.4) (1.24.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy==3.2.4) (2021.10.8)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy==3.2.4) (2.10)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy==3.2.4) (3.0.4)\n",
            "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from jinja2->spacy==3.2.4) (2.0.1)\n",
            "Collecting ru-core-news-sm==3.2.0\n",
            "  Downloading https://github.com/explosion/spacy-models/releases/download/ru_core_news_sm-3.2.0/ru_core_news_sm-3.2.0-py3-none-any.whl (16.4 MB)\n",
            "\u001b[K     |████████████████████████████████| 16.4 MB 46 kB/s \n",
            "\u001b[?25hCollecting pymorphy2>=0.9\n",
            "  Downloading pymorphy2-0.9.1-py3-none-any.whl (55 kB)\n",
            "\u001b[K     |████████████████████████████████| 55 kB 2.5 MB/s \n",
            "\u001b[?25hRequirement already satisfied: spacy<3.3.0,>=3.2.0 in /usr/local/lib/python3.7/dist-packages (from ru-core-news-sm==3.2.0) (3.2.4)\n",
            "Collecting dawg-python>=0.7.1\n",
            "  Downloading DAWG_Python-0.7.2-py2.py3-none-any.whl (11 kB)\n",
            "Collecting pymorphy2-dicts-ru<3.0,>=2.4\n",
            "  Downloading pymorphy2_dicts_ru-2.4.417127.4579844-py2.py3-none-any.whl (8.2 MB)\n",
            "\u001b[K     |████████████████████████████████| 8.2 MB 12.7 MB/s \n",
            "\u001b[?25hRequirement already satisfied: docopt>=0.6 in /usr/local/lib/python3.7/dist-packages (from pymorphy2>=0.9->ru-core-news-sm==3.2.0) (0.6.2)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (3.0.6)\n",
            "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (3.3.0)\n",
            "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (1.21.5)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (1.8.2)\n",
            "Requirement already satisfied: thinc<8.1.0,>=8.0.12 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (8.0.15)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (2.11.3)\n",
            "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (1.0.2)\n",
            "Requirement already satisfied: typer<0.5.0,>=0.3.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (0.4.1)\n",
            "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (2.0.7)\n",
            "Requirement already satisfied: click<8.1.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (7.1.2)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (1.0.6)\n",
            "Requirement already satisfied: wasabi<1.1.0,>=0.8.1 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (0.9.1)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (21.3)\n",
            "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.8 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (3.0.9)\n",
            "Requirement already satisfied: typing-extensions<4.0.0.0,>=3.7.4 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (3.10.0.2)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (4.64.0)\n",
            "Requirement already satisfied: srsly<3.0.0,>=2.4.1 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (2.4.3)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (2.23.0)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (2.0.6)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (0.4.1)\n",
            "Requirement already satisfied: pathy>=0.3.5 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (0.6.1)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (57.4.0)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from catalogue<2.1.0,>=2.0.6->spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (3.8.0)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=20.0->spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (3.0.8)\n",
            "Requirement already satisfied: smart-open<6.0.0,>=5.0.0 in /usr/local/lib/python3.7/dist-packages (from pathy>=0.3.5->spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (5.2.1)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (3.0.4)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (2.10)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (1.24.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (2021.10.8)\n",
            "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from jinja2->spacy<3.3.0,>=3.2.0->ru-core-news-sm==3.2.0) (2.0.1)\n",
            "Installing collected packages: pymorphy2-dicts-ru, dawg-python, pymorphy2, ru-core-news-sm\n",
            "Successfully installed dawg-python-0.7.2 pymorphy2-0.9.1 pymorphy2-dicts-ru-2.4.417127.4579844 ru-core-news-sm-3.2.0\n",
            "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
            "You can now load the package via spacy.load('ru_core_news_sm')\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Затем, как и все библиотеки, библиотеку нужно импортировать."
      ],
      "metadata": {
        "id": "HPu8TT4-tNy9"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3lDZ6oXGjjTI"
      },
      "outputs": [],
      "source": [
        "import spacy"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Загрузка необходимой языковой модели:\n"
      ],
      "metadata": {
        "id": "tGV2F9MmtqBh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nlp = spacy.load(\"ru_core_news_sm\")"
      ],
      "metadata": {
        "id": "pAhYgCpzwIQW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "nlp"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FSV3iUgByzO6",
        "outputId": "7eb03db7-1f41-449e-d868-f8c9c7dfe58e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<spacy.lang.ru.Russian at 0x7f2f716cbf50>"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "В переменной nlp находится объект Language, у которого есть несколько встроенных функций."
      ],
      "metadata": {
        "id": "3vF7lZ3OzI5p"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Первые шаги"
      ],
      "metadata": {
        "id": "Sv6ipQLUzcBX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text = 'Колобок полежал-полежал, да вдруг и покатился — с окна на лавку, с лавки на пол, по полу да к дверям, перепрыгнул через порог в сени, из сеней на крыльцо, с крыльца на двор, со двора за ворота, дальше и дальше.'\n",
        "doc = nlp(text)"
      ],
      "metadata": {
        "id": "WDa3s7pszH1e"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "type(doc)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9klENXn66PSl",
        "outputId": "084b9f7c-1e3c-4c87-b622-dacccd2e8cd2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "spacy.tokens.doc.Doc"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Что такое переменная **doc**? Это объект типа Doc, т.е. набор токенов, который содержит как исходный текст, так и всё, что обработала библиотека spaCy: леммы, именованные сущности, векторы слов и проч."
      ],
      "metadata": {
        "id": "es077NhH5il_"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Токенизация"
      ],
      "metadata": {
        "id": "3K8IfthbzhQI"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Токенизация** - процесс разделения письменного языка на предложения-компоненты (слова и не-слова, типа знаков препинания) - т.е. на **токены**. "
      ],
      "metadata": {
        "id": "eTlzNjSD7C9E"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for token in doc:\n",
        "  print(token.text)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PBcvSc4X7Wdh",
        "outputId": "d78b4c8c-787d-4328-a46f-809328fbfc2f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Колобок\n",
            "полежал\n",
            "-\n",
            "полежал\n",
            ",\n",
            "да\n",
            "вдруг\n",
            "и\n",
            "покатился\n",
            "—\n",
            "с\n",
            "окна\n",
            "на\n",
            "лавку\n",
            ",\n",
            "с\n",
            "лавки\n",
            "на\n",
            "пол\n",
            ",\n",
            "по\n",
            "полу\n",
            "да\n",
            "к\n",
            "дверям\n",
            ",\n",
            "перепрыгнул\n",
            "через\n",
            "порог\n",
            "в\n",
            "сени\n",
            ",\n",
            "из\n",
            "сеней\n",
            "на\n",
            "крыльцо\n",
            ",\n",
            "с\n",
            "крыльца\n",
            "на\n",
            "двор\n",
            ",\n",
            "со\n",
            "двора\n",
            "за\n",
            "ворота\n",
            ",\n",
            "дальше\n",
            "и\n",
            "дальше\n",
            ".\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Выведем общее количество токенов в тексте:"
      ],
      "metadata": {
        "id": "2zoUuv698Zyn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"Всего в тексте {} токенов\".format(len(doc)))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9NSfXGtG8c9f",
        "outputId": "30b27ff2-048e-4f94-e6f7-3e7f472502be"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Всего в тексте 51 токенов\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Очистим текст от знаков препинания и стоп-слов (незначительных слов, которые часто встречаются в тексте)"
      ],
      "metadata": {
        "id": "YZbn0a-H_XpC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "clean_doc = []\n",
        "for token in doc:\n",
        "  if not token.is_stop and not token.is_punct:\n",
        "    clean_doc.append(token)\n",
        "print(clean_doc)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wXxhTvp1_fSv",
        "outputId": "e5c4cad8-7bf7-4943-b98a-1198e09d65e2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[Колобок, полежал, полежал, вдруг, покатился, окна, лавку, лавки, пол, полу, дверям, перепрыгнул, через, порог, сени, сеней, крыльцо, крыльца, двор, со, двора, ворота, дальше, дальше]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#альтернативная запись кода сверху\n",
        "# clean_doc=[token for token in doc if not token.is_stop and not token.is_punct]"
      ],
      "metadata": {
        "id": "dGyb2XYIASzH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"В очищенном тексте {} токенов\".format(len(clean_doc)))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0gxcIEMU_uvp",
        "outputId": "709e63dd-fdf0-45c8-e0cf-e9a81563771c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "В очищенном тексте 24 токенов\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Стоит отметить, что каждый токен, находящийся в полученном очищенном массиве, является уникальным, так как содержит свою индивидуальную информацию."
      ],
      "metadata": {
        "id": "zTYyEhYBADBw"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Лемматизация"
      ],
      "metadata": {
        "id": "BVQNaFVb7jmR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Лемматизация** - приведение слов к начальной форме (отдельные слова - **леммы**)."
      ],
      "metadata": {
        "id": "po9Gwvto7v0r"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for token in doc:\n",
        "  print(token.lemma_)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Om2ElQwn7Zgv",
        "outputId": "69864c5d-c770-44f3-efba-0dc91c8ebcd2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "колобок\n",
            "полежать\n",
            "-\n",
            "полежать\n",
            ",\n",
            "да\n",
            "вдруг\n",
            "и\n",
            "покатиться\n",
            "—\n",
            "с\n",
            "окно\n",
            "на\n",
            "лавка\n",
            ",\n",
            "с\n",
            "лавка\n",
            "на\n",
            "пол\n",
            ",\n",
            "по\n",
            "пол\n",
            "да\n",
            "к\n",
            "дверь\n",
            ",\n",
            "перепрыгнуть\n",
            "через\n",
            "порог\n",
            "в\n",
            "сень\n",
            ",\n",
            "из\n",
            "сеней\n",
            "на\n",
            "крыльцо\n",
            ",\n",
            "с\n",
            "крыльцо\n",
            "на\n",
            "двор\n",
            ",\n",
            "со\n",
            "двор\n",
            "за\n",
            "ворота\n",
            ",\n",
            "дальше\n",
            "и\n",
            "дальше\n",
            ".\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Вывод частей речи"
      ],
      "metadata": {
        "id": "m8sIoa_F8wB3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for token in doc:\n",
        "  print(token.text,'---- ',token.pos_)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cNyLX4sQ8zH2",
        "outputId": "864c6024-0c6c-48c9-cf08-22a7bea82d42"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Колобок ----  PROPN\n",
            "полежал ----  VERB\n",
            "- ----  NOUN\n",
            "полежал ----  VERB\n",
            ", ----  PUNCT\n",
            "да ----  CCONJ\n",
            "вдруг ----  ADV\n",
            "и ----  PART\n",
            "покатился ----  VERB\n",
            "— ----  PUNCT\n",
            "с ----  ADP\n",
            "окна ----  NOUN\n",
            "на ----  ADP\n",
            "лавку ----  NOUN\n",
            ", ----  PUNCT\n",
            "с ----  ADP\n",
            "лавки ----  NOUN\n",
            "на ----  ADP\n",
            "пол ----  NOUN\n",
            ", ----  PUNCT\n",
            "по ----  ADP\n",
            "полу ----  NOUN\n",
            "да ----  PART\n",
            "к ----  ADP\n",
            "дверям ----  NOUN\n",
            ", ----  PUNCT\n",
            "перепрыгнул ----  VERB\n",
            "через ----  ADP\n",
            "порог ----  NOUN\n",
            "в ----  ADP\n",
            "сени ----  NOUN\n",
            ", ----  PUNCT\n",
            "из ----  ADP\n",
            "сеней ----  NOUN\n",
            "на ----  ADP\n",
            "крыльцо ----  NOUN\n",
            ", ----  PUNCT\n",
            "с ----  ADP\n",
            "крыльца ----  NOUN\n",
            "на ----  ADP\n",
            "двор ----  NOUN\n",
            ", ----  PUNCT\n",
            "со ----  ADP\n",
            "двора ----  NOUN\n",
            "за ----  ADP\n",
            "ворота ----  NOUN\n",
            ", ----  PUNCT\n",
            "дальше ----  ADV\n",
            "и ----  CCONJ\n",
            "дальше ----  ADV\n",
            ". ----  PUNCT\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Для определения обозначений можно использовать этот сайт: https://universaldependencies.org/u/pos/\n",
        "Также можно попросить spaCy объяснить, что значит то или иное обозначение:"
      ],
      "metadata": {
        "id": "Z6aX28LI9hgd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "spacy.explain('ADP')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "HMvGHN219jpu",
        "outputId": "90110522-a2ec-40ae-9cf7-1271008f77b5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'adposition'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Выделение именованных сущностей (NER)"
      ],
      "metadata": {
        "id": "Invqm6-HB7qo"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Именованные сущности** - имена людей, названия компаний и проч."
      ],
      "metadata": {
        "id": "3tKNXnViCskM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text = \"Компания 'Яблоко' представила свою новую разработку Илону Маску\"\n",
        "doc2 = nlp(text)\n",
        "print(doc2.ents)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iredH87PC0aO",
        "outputId": "7bf4339b-9d80-4f0f-d03f-22696f1b2904"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "('Яблоко', Илону Маску)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Также можно посмотреть, к какому классу относятся выделенные именованные сущности."
      ],
      "metadata": {
        "id": "AwWylUDRGEZW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for entity in doc2.ents:\n",
        "  print(entity.text,'--- ',entity.label_)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mt_-cDKqGKvC",
        "outputId": "d9a91b74-93f7-4930-f059-4ffc2c17950e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "'Яблоко' ---  ORG\n",
            "Илону Маску ---  PER\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Чтобы выделить именованные сущности в тексте, существует дополнение displacy"
      ],
      "metadata": {
        "id": "PYz-uoajGPyF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from spacy import displacy\n",
        "displacy.render(doc2, style='ent', jupyter=True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "id": "TGXbXTW7GbG2",
        "outputId": "236d0230-18b5-4e14-f64d-7dd1b0c116b6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">Компания \n",
              "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
              "    'Яблоко'\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
              "</mark>\n",
              " представила свою новую разработку \n",
              "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
              "    Илону Маску\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PER</span>\n",
              "</mark>\n",
              "</div></span>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Визуализация деревьев зависимостей"
      ],
      "metadata": {
        "id": "_Emqmg5TipG9"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "При помощи spaCy можно также визуализировать дерево зависимостей, где показаны части речи и части предложения, а также отношение зависимостей."
      ],
      "metadata": {
        "id": "giKjujeiiuuD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from spacy import displacy\n",
        "text= \"Компания 'Яблоко' представила свою новую разработку Илону Маску\"\n",
        "doc3 = nlp(text)\n",
        "\n",
        "displacy.render(doc3,style='dep',jupyter=True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 508
        },
        "id": "kJq5qdmBjF70",
        "outputId": "025d4360-98ba-4c65-9620-6f72a2789cd1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "<span class=\"tex2jax_ignore\"><svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xml:lang=\"ru\" id=\"e67037a00f3b4ae4992a75845d624765-0\" class=\"displacy\" width=\"1450\" height=\"487.0\" direction=\"ltr\" style=\"max-width: none; height: 487.0px; color: #000000; background: #ffffff; font-family: Arial; direction: ltr\">\n",
              "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
              "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">Компания '</tspan>\n",
              "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">NOUN</tspan>\n",
              "</text>\n",
              "\n",
              "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
              "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"225\">Яблоко'</tspan>\n",
              "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"225\">NOUN</tspan>\n",
              "</text>\n",
              "\n",
              "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
              "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"400\">представила</tspan>\n",
              "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"400\">VERB</tspan>\n",
              "</text>\n",
              "\n",
              "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
              "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"575\">свою</tspan>\n",
              "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"575\">DET</tspan>\n",
              "</text>\n",
              "\n",
              "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
              "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"750\">новую</tspan>\n",
              "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"750\">ADJ</tspan>\n",
              "</text>\n",
              "\n",
              "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
              "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"925\">разработку</tspan>\n",
              "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"925\">NOUN</tspan>\n",
              "</text>\n",
              "\n",
              "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
              "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1100\">Илону</tspan>\n",
              "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1100\">PROPN</tspan>\n",
              "</text>\n",
              "\n",
              "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
              "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1275\">Маску</tspan>\n",
              "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1275\">PROPN</tspan>\n",
              "</text>\n",
              "\n",
              "<g class=\"displacy-arrow\">\n",
              "    <path class=\"displacy-arc\" id=\"arrow-e67037a00f3b4ae4992a75845d624765-0-0\" stroke-width=\"2px\" d=\"M70,352.0 C70,177.0 390.0,177.0 390.0,352.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
              "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
              "        <textPath xlink:href=\"#arrow-e67037a00f3b4ae4992a75845d624765-0-0\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">nsubj</textPath>\n",
              "    </text>\n",
              "    <path class=\"displacy-arrowhead\" d=\"M70,354.0 L62,342.0 78,342.0\" fill=\"currentColor\"/>\n",
              "</g>\n",
              "\n",
              "<g class=\"displacy-arrow\">\n",
              "    <path class=\"displacy-arc\" id=\"arrow-e67037a00f3b4ae4992a75845d624765-0-1\" stroke-width=\"2px\" d=\"M70,352.0 C70,264.5 210.0,264.5 210.0,352.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
              "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
              "        <textPath xlink:href=\"#arrow-e67037a00f3b4ae4992a75845d624765-0-1\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">appos</textPath>\n",
              "    </text>\n",
              "    <path class=\"displacy-arrowhead\" d=\"M210.0,354.0 L218.0,342.0 202.0,342.0\" fill=\"currentColor\"/>\n",
              "</g>\n",
              "\n",
              "<g class=\"displacy-arrow\">\n",
              "    <path class=\"displacy-arc\" id=\"arrow-e67037a00f3b4ae4992a75845d624765-0-2\" stroke-width=\"2px\" d=\"M595,352.0 C595,177.0 915.0,177.0 915.0,352.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
              "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
              "        <textPath xlink:href=\"#arrow-e67037a00f3b4ae4992a75845d624765-0-2\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">det</textPath>\n",
              "    </text>\n",
              "    <path class=\"displacy-arrowhead\" d=\"M595,354.0 L587,342.0 603,342.0\" fill=\"currentColor\"/>\n",
              "</g>\n",
              "\n",
              "<g class=\"displacy-arrow\">\n",
              "    <path class=\"displacy-arc\" id=\"arrow-e67037a00f3b4ae4992a75845d624765-0-3\" stroke-width=\"2px\" d=\"M770,352.0 C770,264.5 910.0,264.5 910.0,352.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
              "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
              "        <textPath xlink:href=\"#arrow-e67037a00f3b4ae4992a75845d624765-0-3\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">amod</textPath>\n",
              "    </text>\n",
              "    <path class=\"displacy-arrowhead\" d=\"M770,354.0 L762,342.0 778,342.0\" fill=\"currentColor\"/>\n",
              "</g>\n",
              "\n",
              "<g class=\"displacy-arrow\">\n",
              "    <path class=\"displacy-arc\" id=\"arrow-e67037a00f3b4ae4992a75845d624765-0-4\" stroke-width=\"2px\" d=\"M420,352.0 C420,89.5 920.0,89.5 920.0,352.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
              "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
              "        <textPath xlink:href=\"#arrow-e67037a00f3b4ae4992a75845d624765-0-4\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">obj</textPath>\n",
              "    </text>\n",
              "    <path class=\"displacy-arrowhead\" d=\"M920.0,354.0 L928.0,342.0 912.0,342.0\" fill=\"currentColor\"/>\n",
              "</g>\n",
              "\n",
              "<g class=\"displacy-arrow\">\n",
              "    <path class=\"displacy-arc\" id=\"arrow-e67037a00f3b4ae4992a75845d624765-0-5\" stroke-width=\"2px\" d=\"M420,352.0 C420,2.0 1100.0,2.0 1100.0,352.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
              "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
              "        <textPath xlink:href=\"#arrow-e67037a00f3b4ae4992a75845d624765-0-5\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">iobj</textPath>\n",
              "    </text>\n",
              "    <path class=\"displacy-arrowhead\" d=\"M1100.0,354.0 L1108.0,342.0 1092.0,342.0\" fill=\"currentColor\"/>\n",
              "</g>\n",
              "\n",
              "<g class=\"displacy-arrow\">\n",
              "    <path class=\"displacy-arc\" id=\"arrow-e67037a00f3b4ae4992a75845d624765-0-6\" stroke-width=\"2px\" d=\"M1120,352.0 C1120,264.5 1260.0,264.5 1260.0,352.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
              "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
              "        <textPath xlink:href=\"#arrow-e67037a00f3b4ae4992a75845d624765-0-6\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">flat:name</textPath>\n",
              "    </text>\n",
              "    <path class=\"displacy-arrowhead\" d=\"M1260.0,354.0 L1268.0,342.0 1252.0,342.0\" fill=\"currentColor\"/>\n",
              "</g>\n",
              "</svg></span>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Описание синтаксических отношений между членами предложения можно посмотреть по ссылке: https://universaldependencies.org/en/dep/"
      ],
      "metadata": {
        "id": "4le7KyI2jhA8"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Распознавание эл. почты"
      ],
      "metadata": {
        "id": "xRQYTi4mCA3F"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "С помощью spaCy также можно распознавать адреса электронной почты:"
      ],
      "metadata": {
        "id": "zaeRcDCrCl4N"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text = \"Готовые работы присылайте на myname@hse.ru\"\n",
        "doc4 = nlp(text)\n",
        "for token in doc4:\n",
        "  if token.like_email:\n",
        "    print(token.text)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kcD2f3TCAhw_",
        "outputId": "f5e7448d-87c5-4030-a29f-ab2e8afe2bd5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "myname@hse.ru\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 2. spaCy для решения некоторых задач."
      ],
      "metadata": {
        "id": "KNHg0Orvb-Or"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Чтение файлов\n",
        "\n",
        "(не связано с работой библиотеки spaCy)"
      ],
      "metadata": {
        "id": "SXqv_jPvfbOB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "file = open(\"jack.txt\", \"r\")\n",
        "text = file.read() # в переменной text будет храниться текст файла. с этим текстом будем потом работать.\n",
        "file.close()\n",
        "\n",
        "print(text)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nb8HOdWVcHbw",
        "outputId": "6ce16b8c-e744-4b2b-e681-73e0f2254314"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Вот дом,\n",
            "Который построил Джек.\n",
            "\n",
            "А это пшеница,\n",
            "Которая в тёмном чулане хранится\n",
            "В доме,\n",
            "Который построил Джек.\n",
            "\n",
            "А это весёлая птица-синица,\n",
            "Которая часто ворует пшеницу,\n",
            "Которая в тёмном чулане хранится\n",
            "В доме,\n",
            "Который построил Джек.\n",
            "\n",
            "Вот кот,\n",
            "Который пугает и ловит синицу,\n",
            "Которая часто ворует пшеницу,\n",
            "Которая в тёмном чулане хранится\n",
            "В доме,\n",
            "Который построил Джек.\n",
            "\n",
            "Вот пёс без хвоста,\n",
            "Который за шиворот треплет кота,\n",
            "Который пугает и ловит синицу,\n",
            "Которая часто ворует пшеницу,\n",
            "Которая в тёмном чулане хранится\n",
            "В доме,\n",
            "Который построил Джек.\n",
            "\n",
            "А это корова безрогая,\n",
            "Лягнувшая старого пса без хвоста,\n",
            "Который за шиворот треплет кота,\n",
            "Который пугает и ловит синицу,\n",
            "Которая часто ворует пшеницу,\n",
            "Которая в тёмном чулане хранится\n",
            "В доме,\n",
            "Который построил Джек.\n",
            "\n",
            "А это старушка, седая и строгая,\n",
            "Которая доит корову безрогую,\n",
            "Лягнувшую старого пса без хвоста,\n",
            "Который за шиворот треплет кота,\n",
            "Который пугает и ловит синицу,\n",
            "Которая часто ворует пшеницу,\n",
            "Которая в тёмном чулане хранится\n",
            "В доме,\n",
            "Который построил Джек.\n",
            "\n",
            "А это ленивый и толстый пастух,\n",
            "Который бранится с коровницей строгою,\n",
            "Которая доит корову безрогую,\n",
            "Лягнувшую старого пса без хвоста,\n",
            "Который за шиворот треплет кота,\n",
            "Который пугает и ловит синицу,\n",
            "Которая часто ворует пшеницу,\n",
            "Которая в тёмном чулане хранится\n",
            "В доме,\n",
            "Который построил Джек.\n",
            "\n",
            "Вот два петуха,\n",
            "Которые будят того пастуха,\n",
            "Который бранится с коровницей строгою,\n",
            "Которая доит корову безрогую,\n",
            "Лягнувшую старого пса без хвоста,\n",
            "Который за шиворот треплет кота,\n",
            "Который пугает и ловит синицу,\n",
            "Которая часто ворует пшеницу,\n",
            "Которая в тёмном чулане хранится\n",
            "В доме,\n",
            "Который построил Джек.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Анонимизация/Маскирование"
      ],
      "metadata": {
        "id": "KtGIghmfojTi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text = \"Только что министр финансов США Джанет Йеллен прочитала нотацию странам, которые 'сидят на двух стульях'\"\n",
        "doc = nlp(text)"
      ],
      "metadata": {
        "id": "lcSIi1bmoc3V"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def update_article(text):\n",
        "  doc = nlp(text)\n",
        "  for word in doc:\n",
        "      if word.ent_type_ =='PER' or word.ent_type_=='ORG' or word.ent_type_=='GPE' or word.ent_type_=='LOC':\n",
        "        text = text.replace(word.text, 'UNKNOWN')\n",
        "  return text"
      ],
      "metadata": {
        "id": "_UGhsmv57BFS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "update_article(text)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "jQudZAtH7dpz",
        "outputId": "d59759ca-46a3-477d-c423-7631ccf0838c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "\"Только что министр финансов UNKNOWN UNKNOWN UNKNOWN прочитала нотацию странам, которые 'сидят на двух стульях'\""
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Схожесть текстов"
      ],
      "metadata": {
        "id": "DNajK1S1sk_F"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Напомним, что у каждого слова есть свое векторное представление. Т.к. векторы представляют собой численное представление, то они используются для различных NLP задач (н-р, для классификации).\n",
        "\n",
        "Слова, близкие по смыслу, будут иметь близкие векторные репрезентации."
      ],
      "metadata": {
        "id": "qxYsaltRsn3J"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "*Note: векторы присутствуют только в модели _lg, в модели _sm присутствуют только тензоры, чувствительные к контексту, поэтому точность такой модели будет ниже*"
      ],
      "metadata": {
        "id": "oqqfPCENul3X"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Чтобы проверить, есть ли вектор для слова, используется метод .has_vector"
      ],
      "metadata": {
        "id": "6tq7KfSZwScm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tokens = nlp(\"я люблю собак.\")\n",
        "for token in tokens:\n",
        "  print(token.text ,' ', token.has_vector)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LieSvbRluDEn",
        "outputId": "5f2c6b97-7df2-40df-c416-31582c6814b0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "я   True\n",
            "люблю   True\n",
            "собак   True\n",
            ".   True\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Чтобы получить вектор слова, необходим метод .vector"
      ],
      "metadata": {
        "id": "kpi7iYYkwyr6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tokens = nlp(\"я люблю собак.\")\n",
        "for token in tokens:\n",
        "  print(token.text ,' ', token.vector)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RSDicLq-wg5R",
        "outputId": "a52cbd6b-f828-46b6-fea2-221d18773760"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "я   [ 3.828765   -0.9292036  -0.2817854  -0.7526239   3.9445112   0.3411465\n",
            "  1.6517617   0.90737     2.2479703  -2.7426634   0.02391347 -1.2609829\n",
            "  1.8857346  -0.5423751  -0.5265444   0.08626613  1.9380867  -0.62348616\n",
            "  3.9450188  -0.7833001   0.13577552 -0.8242404   2.5525365   2.4178858\n",
            " -0.47625655 -0.42145932 -0.14687943 -1.7458341  -0.01297763 -1.8612216\n",
            " -0.9800793  -0.83266634 -0.10449647  0.9873901  -0.9411416   1.0138882\n",
            "  0.02437502 -0.5348282   0.6854346  -1.2680278   0.32444894 -0.09298873\n",
            " -0.86737734 -2.1450782  -0.50667995 -0.74722505  0.82992125 -1.387329\n",
            " -0.48365015  1.4036915  -0.23721686  2.5811155   1.0700673  -0.40473026\n",
            "  0.42977425 -0.7654067   0.96398205  0.8982876   0.02117755 -2.2847018\n",
            " -1.3538965  -0.83054876 -0.60932815  1.0414056   0.19321436 -1.6292319\n",
            " -0.55147874  0.65343446  1.1539578  -1.0563307  -1.4022238   0.32088804\n",
            "  0.53807634 -2.0122077  -1.9084041   0.6792588  -1.1888739  -2.4656668\n",
            "  2.418604    0.13217103 -0.801353   -0.11600313  3.0567114   1.4267045\n",
            " -0.3154736   0.8127646   0.99258506 -0.13258454  1.0982122   0.72355354\n",
            " -1.2946806  -1.0401527   2.7380683  -1.3815781  -1.6375496  -0.47055912]\n",
            "люблю   [-0.31233215  1.0525746  -0.0037497   0.1016254  -0.12714477  1.6217134\n",
            "  0.76504314  1.4074327  -0.5023255  -0.3226992   0.9489269   0.2901587\n",
            "  0.3236347  -0.96841836  1.6453195  -1.3638335  -0.10746869 -1.1583714\n",
            "  1.3763728  -1.3810712  -1.23751     1.6364099   1.0003673  -0.81783646\n",
            " -0.46902606  2.6692348  -1.3796422   0.6462704  -0.5726211  -1.148063\n",
            "  1.535435   -0.5636058  -0.38561362 -0.5731494  -1.217347   -0.40871382\n",
            "  0.68293476 -0.9678803   2.5351887   2.7093515  -0.2454313  -1.8950778\n",
            "  1.2786462  -1.3288074  -1.6062586  -0.6904209   0.2526326  -0.06722075\n",
            " -0.56883955  0.01570787  0.00889659  0.57183355  0.69208026 -0.84386015\n",
            " -0.01983777  1.5448449   0.42652595 -0.25373656 -1.039212   -2.3854303\n",
            " -0.0222826   1.7412778  -1.5726726  -0.38433802 -0.20522213 -1.3279716\n",
            "  0.32179105 -0.25120658 -1.35171    -0.59117544  1.2125425   1.1182358\n",
            " -0.9971988  -0.46455473  2.4029074  -1.2326207   0.65041673  0.85756415\n",
            " -0.65758955 -0.61363345 -0.4491722  -0.35140145  0.00714359 -0.19285989\n",
            "  0.42436624 -1.2340128  -0.5006565  -0.8637799  -0.3044531   0.790759\n",
            " -1.5545516  -0.90118057  0.71700215 -0.7165414   0.4709493   2.461989  ]\n",
            "собак   [ 0.8245688   0.43352473 -0.5611302  -1.0606189   0.9778094  -1.3948038\n",
            " -0.4742537   0.62659985 -0.9719096   3.0300179  -0.59660596  1.2315724\n",
            " -0.25917915  0.7646779   1.0624065   0.7725772   0.01661268 -0.9772379\n",
            " -1.6123146   0.1243805  -1.5479795  -0.3143524   0.6948588   0.10987012\n",
            "  0.7664566   0.19780502  0.49608606 -0.96360916 -0.17423064 -0.3634851\n",
            "  0.1966189  -0.3348161  -0.88590586  0.70826507 -1.4182962   0.45733115\n",
            " -0.3918749  -1.5569553   0.3204189   0.09068728  1.5913812  -0.59535605\n",
            " -0.1807113  -0.5174579  -1.84846     2.0945902   1.8010666  -2.4855728\n",
            "  1.2803816   0.05928841  1.227521    0.8750104   0.9997435   1.3918396\n",
            "  0.85676146  2.7283862  -0.6648061   1.2069917  -1.1547637  -0.50426185\n",
            " -0.40680882 -0.21892105 -2.5515401  -0.31041437  1.9856839   1.6340394\n",
            " -0.01631918  1.9750248  -1.3316686  -0.41622826  0.5378294   0.5130491\n",
            "  0.49994594 -0.44633842 -0.84818906 -2.148425    0.34698737  0.80199695\n",
            " -0.02559967 -1.5663371  -0.07171977 -0.23658264  0.74135506 -0.18598205\n",
            " -0.93592334  0.4478758  -0.522435   -2.9451745   0.62387145 -0.02736606\n",
            " -0.22797742  0.26699162 -1.1730536   1.5164773  -0.69920135 -0.79150534]\n",
            ".   [-0.35366908 -0.5015502   3.5014632   0.9411947   0.16439357 -1.1684\n",
            " -0.6362097  -0.2878844  -0.0714331  -1.7924914   0.5807006   0.5367167\n",
            " -0.34639436 -0.50613797  0.22740781 -0.67027116  1.7487576  -0.15405056\n",
            " -0.51808876 -0.84195334  0.13812837 -1.8942225   1.5613415  -1.1223156\n",
            " -0.14968534  1.1225891   2.111421   -0.22518766  0.8179241  -1.7671742\n",
            " -0.00740322  5.5130534   0.12650192  0.13924399 -0.45080325 -0.5360921\n",
            " -0.47923473  3.819569   -0.05781966 -1.3730958   0.09233567 -0.9691987\n",
            "  1.185712   -0.59252703 -0.9245212  -0.7673321  -1.06269    -0.5470942\n",
            "  3.705947    0.8222507  -0.19562116  0.5871351  -0.18152162 -0.01348343\n",
            " -1.5090492  -0.42254362 -0.32557526 -0.36665577  0.17684318 -0.38929945\n",
            " -0.5753226  -0.79269505 -1.3175877  -1.6445515  -0.82380867 -0.6053721\n",
            " -1.4542451  -0.21520643 -0.3023388   0.28181264  0.3548311   0.47766113\n",
            " -0.40048587 -0.5462736   2.820611    2.3719845   0.40145308  0.2810073\n",
            " -0.36825252 -1.543912    0.72907907 -0.47069246 -0.6483437  -0.38581485\n",
            " -0.91550046  3.0900805  -1.1538444  -2.2322192   2.4885626   0.0879325\n",
            " -0.35555148 -0.2731576  -0.2112396  -1.6341829   1.8624469  -0.52237225]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Можно также взять L2 норму векторов с помощью метода .vector_norm"
      ],
      "metadata": {
        "id": "yQVmLU5S5wd7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for token in tokens:\n",
        "  print(token.text ,' ', token.vector_norm)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YIJBUu2Bw1fz",
        "outputId": "5cf655b0-9122-41b4-c47a-84e5122412db"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "я   13.860122\n",
            "люблю   10.697362\n",
            "собак   10.900073\n",
            ".   12.992036\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Как найти **сходство двух токенов**? Для этого используется функция **similarity()**."
      ],
      "metadata": {
        "id": "7U_S8ixd59EW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "token_1=nlp(\"плохо\")\n",
        "token_2=nlp(\"ужасно\")\n",
        "\n",
        "similarity_score=token_1.similarity(token_2)\n",
        "print(similarity_score)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UJumksqv6JbA",
        "outputId": "3f927d32-fbfe-48d3-f290-6a243bc6cf0f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.8359503163356972\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:4: UserWarning: [W007] The model you're using has no word vectors loaded, so the result of the Doc.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
            "  after removing the cwd from sys.path.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "token_1=nlp(\"плохо\")\n",
        "token_2=nlp(\"хорошо\")\n",
        "\n",
        "similarity_score=token_1.similarity(token_2)\n",
        "print(similarity_score)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3GujVROj8cD4",
        "outputId": "319c0763-83be-4195-c222-c5887f9e9ee6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.8924345789883013\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:4: UserWarning: [W007] The model you're using has no word vectors loaded, so the result of the Doc.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
            "  after removing the cwd from sys.path.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "review_1=nlp('Еда была неплохая.')\n",
        "review_2=nlp('Еда была хорошая.')\n",
        "review_3=nlp('Мне не понравилась еда.')\n",
        "review_4=nlp('Еда была отвратительная.')\n",
        "\n",
        "score_1=review_1.similarity(review_2)\n",
        "print('Сходство между ревью 1 и 2',score_1)\n",
        "\n",
        "score_2=review_3.similarity(review_4)\n",
        "print('Сходство между ревью 3 и 4',score_2)\n",
        "\n",
        "score_3=review_2.similarity(review_4)\n",
        "print('Сходство между ревью 2 и 4',score_3)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HULPDmx08jN0",
        "outputId": "a3a034fe-831d-4525-cef0-ec801ca90ac0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Сходство между ревью 1 и 2 0.936561469073574\n",
            "Сходство между ревью 3 и 4 0.5653143774231582\n",
            "Сходство между ревью 2 и 4 0.9590623363437175\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:6: UserWarning: [W007] The model you're using has no word vectors loaded, so the result of the Doc.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
            "  \n",
            "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:9: UserWarning: [W007] The model you're using has no word vectors loaded, so the result of the Doc.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
            "  if __name__ == '__main__':\n",
            "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:12: UserWarning: [W007] The model you're using has no word vectors loaded, so the result of the Doc.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
            "  if sys.path[0] == '':\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Обратите внимание на **UserWarning** message: так как в модели, которую мы используем, нет векторов, метод нахождения сходства будет использовать другие данные. Поэтому для более точного нахождения сходства загрузим другую модель."
      ],
      "metadata": {
        "id": "RJSjART16QHv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!python -m spacy download ru_core_news_lg"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lVy3nFcU6MXy",
        "outputId": "2257cae2-7cb7-4108-93b7-4d121faf0f65"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting ru-core-news-lg==3.2.0\n",
            "  Downloading https://github.com/explosion/spacy-models/releases/download/ru_core_news_lg-3.2.0/ru_core_news_lg-3.2.0-py3-none-any.whl (514.5 MB)\n",
            "\u001b[K     |████████████████████████████████| 514.5 MB 5.4 kB/s \n",
            "\u001b[?25hRequirement already satisfied: pymorphy2>=0.9 in /usr/local/lib/python3.7/dist-packages (from ru-core-news-lg==3.2.0) (0.9.1)\n",
            "Requirement already satisfied: spacy<3.3.0,>=3.2.0 in /usr/local/lib/python3.7/dist-packages (from ru-core-news-lg==3.2.0) (3.2.4)\n",
            "Requirement already satisfied: pymorphy2-dicts-ru<3.0,>=2.4 in /usr/local/lib/python3.7/dist-packages (from pymorphy2>=0.9->ru-core-news-lg==3.2.0) (2.4.417127.4579844)\n",
            "Requirement already satisfied: docopt>=0.6 in /usr/local/lib/python3.7/dist-packages (from pymorphy2>=0.9->ru-core-news-lg==3.2.0) (0.6.2)\n",
            "Requirement already satisfied: dawg-python>=0.7.1 in /usr/local/lib/python3.7/dist-packages (from pymorphy2>=0.9->ru-core-news-lg==3.2.0) (0.7.2)\n",
            "Requirement already satisfied: typer<0.5.0,>=0.3.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (0.4.1)\n",
            "Requirement already satisfied: typing-extensions<4.0.0.0,>=3.7.4 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (3.10.0.2)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (2.11.3)\n",
            "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (1.21.5)\n",
            "Requirement already satisfied: wasabi<1.1.0,>=0.8.1 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (0.9.1)\n",
            "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (1.0.2)\n",
            "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (2.0.7)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (0.4.1)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (2.23.0)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (1.8.2)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (21.3)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (57.4.0)\n",
            "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.8 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (3.0.9)\n",
            "Requirement already satisfied: srsly<3.0.0,>=2.4.1 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (2.4.3)\n",
            "Requirement already satisfied: click<8.1.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (7.1.2)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (1.0.6)\n",
            "Requirement already satisfied: thinc<8.1.0,>=8.0.12 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (8.0.15)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (3.0.6)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (2.0.6)\n",
            "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (3.3.0)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (4.64.0)\n",
            "Requirement already satisfied: pathy>=0.3.5 in /usr/local/lib/python3.7/dist-packages (from spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (0.6.1)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from catalogue<2.1.0,>=2.0.6->spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (3.8.0)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=20.0->spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (3.0.8)\n",
            "Requirement already satisfied: smart-open<6.0.0,>=5.0.0 in /usr/local/lib/python3.7/dist-packages (from pathy>=0.3.5->spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (5.2.1)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (2021.10.8)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (2.10)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (1.24.3)\n",
            "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from jinja2->spacy<3.3.0,>=3.2.0->ru-core-news-lg==3.2.0) (2.0.1)\n",
            "Installing collected packages: ru-core-news-lg\n",
            "Successfully installed ru-core-news-lg-3.2.0\n",
            "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
            "You can now load the package via spacy.load('ru_core_news_lg')\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "nlp = spacy.load(\"ru_core_news_lg\")"
      ],
      "metadata": {
        "id": "L707siBk6kt6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "token_1=nlp(\"плохо\")\n",
        "token_2=nlp(\"ужасно\")\n",
        "\n",
        "similarity_score=token_1.similarity(token_2)\n",
        "print(similarity_score)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vufn4ZcL7me1",
        "outputId": "87e95df1-90a5-49a8-9e3a-5da95fd19ada"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.4543616228831531\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "token_1=nlp(\"плохо\")\n",
        "token_2=nlp(\"хорошо\")\n",
        "\n",
        "similarity_score=token_1.similarity(token_2)\n",
        "print(similarity_score)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "l_5eY3-k7ti-",
        "outputId": "bb028697-37e1-47bf-9d2e-e6f8a28c93dc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.6755168968764647\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "review_1=nlp('Еда была неплохая.')\n",
        "review_2=nlp('Еда была хорошая.')\n",
        "review_3=nlp('Мне не понравилась еда.')\n",
        "review_4=nlp('Еда была отвратительная.')\n",
        "\n",
        "score_1=review_1.similarity(review_2)\n",
        "print('Сходство между ревью 1 и 2',score_1)\n",
        "\n",
        "score_2=review_3.similarity(review_4)\n",
        "print('Сходство между ревью 3 и 4',score_2)\n",
        "\n",
        "score_3=review_2.similarity(review_4)\n",
        "print('Сходство между ревью 2 и 4',score_3)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wAsM31iJ7vkn",
        "outputId": "ce237157-3fb0-4c0b-8c8b-68177f04b078"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Сходство между ревью 1 и 2 0.9224265232448274\n",
            "Сходство между ревью 3 и 4 0.482127656480746\n",
            "Сходство между ревью 2 и 4 0.7747235859680845\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Нахождение паттернов"
      ],
      "metadata": {
        "id": "4rbjIcaA9QV-"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "С помощью модели spaCy, которая размечает члены предложения, мы можем находить сочетания слов, которые соответствуют заданному нами шаблону токенов. Для этого используют Matcher."
      ],
      "metadata": {
        "id": "nlH1K8Z79T8U"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from spacy.matcher import Matcher \n",
        "matcher = Matcher(nlp.vocab)"
      ],
      "metadata": {
        "id": "yMpQIp1x9uo1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Для начала нам нужно задать шаблон. Выглядеть он будет следующим образом:"
      ],
      "metadata": {
        "id": "bfRTCdmI94Yc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "my_pattern=[{\"POS\": \"PROPN\"}, {\"LIKE_NUM\": True}] "
      ],
      "metadata": {
        "id": "aa1pssxw926E"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Данный паттерн означает, что сначала мы ищем сочетание двух слов, где первое слово - PROPN по POS (имя собственное по части речи), а второе - LIKE_NUM (число). Проверим данный шаблон на каком-нибудь текстовом примере."
      ],
      "metadata": {
        "id": "cIJmi9uX-IWh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "matcher.add(\"VersionFinder\", [my_pattern])"
      ],
      "metadata": {
        "id": "_NwyjT4Q-t6B"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "text = \"Она давно хотела купить IPhone 13, но могла себе позволить только IPhone 11.\"\n",
        "doc = nlp(text)\n",
        "\n",
        "desired_matches = matcher(doc)\n",
        "desired_matches"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QTw0wJG1-bES",
        "outputId": "44cf977a-6f4c-427f-81b1-ece722fe816b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[(6950581368505071052, 4, 6), (6950581368505071052, 12, 14)]"
            ]
          },
          "metadata": {},
          "execution_count": 78
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "В выходных данных видно, что было найдено два попадания под шаблон. В каждом из этих объектов 1 элемент - matcher_id, 2 - начало попадания (первый токен), 3 - конец попадания (последний токен +1).\n",
        "\n",
        "*Помните, что отсчет идет по токенам и с 0 элемента. Последний в данном случае не берется - такие правила срезов.*"
      ],
      "metadata": {
        "id": "dbG1Ugve_Bm3"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Давайте выведем наши попадания."
      ],
      "metadata": {
        "id": "EL7xdZ1t_djg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for i in desired_matches:\n",
        "  match = doc[i[1]:i[2]]\n",
        "  print(match.text)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KZCKsGg3--z5",
        "outputId": "9994b654-ef36-487b-bcdd-f5347e1f23fa"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "IPhone 13\n",
            "IPhone 11\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Попробуем взять другой шаблон, который будет брать сочетание одного слова с любым другим словом. Помним, что у слов есть разные формы, так что стоит взять за шаблон лемму слова."
      ],
      "metadata": {
        "id": "HrwQLo_9ABqR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "my_pattern=[{\"LEMMA\": \"съездить\"}, {\"POS\": \"ADP\"}, {\"POS\": \"PROPN\"}] "
      ],
      "metadata": {
        "id": "kYEwkA5I_4oY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "matcher = Matcher(nlp.vocab)\n",
        "matcher.add(\"VersionFinder\", [my_pattern])"
      ],
      "metadata": {
        "id": "1HOxXIN8AfTg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "text = \"Вы когда успели съездить в Европу? Вот моя дочка съездила в Италию, ей очень понравилось. Но я сама летом съезжу в Египет.\"\n",
        "doc = nlp(text)\n",
        "\n",
        "desired_matches = matcher(doc)\n",
        "desired_matches"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DtYd-f9lAhk-",
        "outputId": "6c22b565-5e06-459a-dbe9-5c9bccdb451b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[(6950581368505071052, 3, 6), (6950581368505071052, 10, 13)]"
            ]
          },
          "metadata": {},
          "execution_count": 90
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "for i in desired_matches:\n",
        "  match = doc[i[1]:i[2]]\n",
        "  print(match.text)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gYoHnOg1Azs-",
        "outputId": "13e4083c-42cd-4faf-b46a-64d5555c7ecf"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "съездить в Европу\n",
            "съездила в Италию\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "58AMH0shBBeO"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}